1. Field of the Invention
The present invention relates to a multi-media document editing system for editing a document composed of multi-media data as elements. More particularly, the invention is concerned with a multi-media document editing system for editing a display position (a spatial structure) of a multi-media data group as well as display start and end (a time-related structure) thereof and reconstructing multi-media data held in a data base, an AV device, or the like, on the basis of the results of the editing.
2. Description of the Prior Art
The multi-media document is constructed as a document including not only such static media data as characters, graphics and static images but also such dynamic media data as voice and dynamic images. In editing the multi-media document, therefore, there usually is conducted an editing operation for designating a display position and a display timing for each multi-media data.
As to the display timing, not only the display sequence of multi-media data in the multi-media document is designated but also a correlation of multi-media data is taken into account. For this reason the designation of the display timing is called synchronous designation. The following two types of methods are available as synchronous designation methods. Further, there is an editing user interface based on each of the methods.
The first method is a synchronous designation method based on time designation. According to this method, a display time of each individual multi-media data is designated explicitly. The second method is a synchronous designation method based on relation designation. According to this method, an interdependence relation of multi-media data is designated.
The first synchronous designation method based on time designation is a method used, for example, in the editing system shown in the literature "IEEE SOFTWARE, Construction Set for multimedia Applications, Vol. 6 No. 1, Jan. 1989, pp. 37-43." According to this designation method, a reproduction time is designated explicitly for each multi-media data. An example of the user interface based on the first synchronous designation method is shown in FIG. 1. In the display screen of the user interface, a rectangle having a width in the X axis direction is proportional to a display time width that is disposed on a time axis extending from the left to the right. This user interface, called the time line interface, is advantageous in that a temporal structure of the multi-media data can be grasped intuitively.
When multi-media data are to be reproduced at the same time in the time line interface, as shown in FIG. 1, both are designated to the same display start time. This is attained by positioning the left end points of the rectangular multi-media data on the time axis exactly at X coordinates on the same time axis. In FIG. 1, VIDEO1, NARRATION1 and SUBTITLE are designated at the same start time.
According to the second synchronous designation method based on relation designation, a temporal structure is expressed on the basis of a correlation of multi-media data. For example, a time relation is expressed in the following manner: "The start event of a certain multi-media occurs after the occurrence of the end event of another element." According to this synchronous designation method, even when the time information of a multi-media data is changed, the influence thereof can be limited to the object adjacent to that multi-media data.
For example, reference is here made to the literature "Specifying Temporal Behavior in Hypermedia Documents, Proceedings of the European Conference on Hypertext '92, Dec. 1992." As introduced in this literature, according to the synchronous designation method based on relation designation, multi-media data is expressed as a segment whose length is the display time width, and display start and end events are each expressed as a circle. Start and end events of the entire document are each expressed as a small rectangle. Further, a correlation of events is expressed in terms of a digraph. In FIG. 2 there is shown an example of user interface display in the synchronous designation method based on such relation designation.
As shown in FIG. 2, on the user interface display are arranged a plurality of segments (multi-media data) each having events at both ends thereof which events are each expressed as a circle. The user designates a relation of multi-media data by stretching a digraph between the events. Here there is no correlation between the spatial positional relation on the user interface and time.
The foregoing first synchronous designation method based on time designation involves a problem such that there is no time relation between multi-media data, so for example when the display time of one of the multi-media data arranged continuously on the time axis is changed, manual modification of the time information of each remaining multi-media data is required and thus the editing operation becomes complicated. For example, when VIDEO3 is to be inserted after VIDEO1 in FIG. 1, the user is required to move VIDEO2 by a distance corresponding to the display time width of VIDEO3 and thereafter dispose VIDEO3. This operation is complicated.
In this connection, in such a time line-based editing interface as shown in FIG. 1, the positions of VIDEO1, VIDEO2 and VIDEO3 represent display times of those elements, so for affecting a desired editing it is necessary to give consideration so that each element is disposed in a predetermined display time position without being superimposed on another. Thus, the user's burden in the editing operation is heavy.
In the case where the synchronism of plural elements is to be designated accurately by the synchronous designation method based on time designation, it is necessary to set the same time for all of the elements. In the example shown in FIG. 1, VIDEO1, NARRATION1 and SUBTITLE are arranged in the position corresponding to the time "3 seconds," but with this tacit synchronism relation kept intact, if the display start time of these multi-media data are to be set to the position corresponding to the time "1 second," it is necessary for the user to make rearrangement so that the left end points of those three multi-media data assume the position corresponding to the time "1 second."
More particularly, it is required that the tacit synchronism relation in the position of time "3 seconds" which has once been assumed by the user be destroyed and the operation for accurate time designation be newly performed for all of the related elements, which operation is troublesome. In the case of operation on the editing user interface, since the editing operation of elements alignment is a manual operation, the possibility of occurrence of an error is high.
The second synchronous designation method based on relation designation also involves a problem such that since there is no concept of time coordinates on the user interface, it is impossible to judge intuitively whether the reproduction timing of each multi-media data is relatively fast or slow. In FIG. 2, for example, which of NARRATION1 and NARRATION2 is to be reproduced first cannot be judged unless the digraph relation is traced.
The synchronous designation method based on relation designation further involves a problem such that it is impossible to collectively perform the operations of selecting plural multi-media data elements and moving them while maintaining their time relation in that range. For example, in FIG. 2, when NARRATION1 and VIDEO1 are to be "moved" behind VIDEO2 while maintaining start synchronism at display start event E1, it is necessary to take the following procedure.
(1) Delete all of the digraphs (those close to START, VIDEO1 and SUBTITLE) associated with the node of display start event E1. PA1 (2) Delete the node of display start event E1. PA1 (3) Delete all of the digraphs (those close to END and SUBTITLE) associated with the node of display end event E3. PA1 (4) Move VIDEO1 to the position where E3 is the start event. PA1 (5) Set the node of display end event E3 as the start event of VIDEO1. PA1 (6) Create a node of display end event E4 as an end event of VIDEO1. PA1 (7) Set a digraph between the node of display end event E3 and NARRATION1. PA1 (8) Set a digraph between the node of display end event E4 and END node. PA1 (9) Set a digraph among the node of display end event E2, START node and display start node of SUBTITLE.
Thus, when a desired editing operation is to be executed in the editing user interface according to the synchronous designation method based on relation designation, it is necessary to follow the above complicated procedure.
Another problem involved in the synchronous designation method based on relation designation is that the start timing of events depends on the reproduction time of multi-media data which is reproduced in advance of the event. For example, as shown in FIG. 2, in the case where there has been made a synchronous designation based on relation designation and the display start event E1 starts at the time "0 second," the start time of display end event E2 is "6 seconds," assuming that the reproduction of VIDEO1 requires 6 seconds. That is, the reproduction start time of VIDEO2 is the time "6 seconds" after the lapse of 6 seconds. If the reproduction start time of VIDEO2 is to be set to the time "10 seconds" while keeping VIDEO1 intact, the user is required to insert a 4-second, signal-free, multi-media data (blank) behind the display end event E2, then set a node of event E5 connected to the display end event of the 4-second, signal-free, multi-media data (blank) and connect the node of event E5 with the display start event of VIDEO2 through a digraph. Thus, the alterations are troublesome like the foregoing editing operation.
Generally, according to the conventional synchronous designation method based on relation designation, the temporal structure editing operation requires setting of both "synchronous events" and a digraph which expresses the relation between them and elements. This involves a drawback that in the case of performing the editing operation while maintaining the existing relation to a specific group of elements, the number of operations increases in proportion to associated elements. Another drawback is that the results of the editing operation are difficult to be judged intuitively on the user interface.